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ABSTRACT 

In March 1991, the College Standards and 
Accreditation Council (CSAC) Establishment Board, created by Canada 1 s 
Minister of Education, proposed wide-ranging reforms for college 
education and training. This paper reviews some of the ways in which 
Algonquin College (AC), in Ontario, is planning to interface and 
articulate with CSAC initiatives to ensure full compliance and retain 
institutional autonomy. First, the CSAC initiatives are described for 
Ontario, indicating that they rely on learning outcomes as the basis 
for the definition of program standards and aim to establish 
system-wide outcome standards by province to ensure similar outcomes 
for similar college programs and demonstrate to the public that 
standards are being met. Next, characteristics of learning outcomes 
and how they are determined are reviewed, and information is 
presented on drafting learning outcomes statements. A curriculum 
model for outcomes-based education is then presented and s^sessment 
and evaluation strategies are discussed, including eight types of 
instruments or methods. Finally, the province-wide regulations for 
program review are described and the program review system at AC is 
detailed, including the use of critical performance indicators, 
cyclical program evaluation, ongoing program appraisal, program 
review, and the program review report. (KP) 
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OVERVIEW OF THE PRESENTATION (1 HOUR, 30 MINUTES) 

Introduction to the Presentation 

Ontario's "Vision" for the Colleges 

"Quality - Access - Accountability" 
"Performance - Results - Accountability" 

The Role of Learning Outcomes 
(focus on performance standards) 

Curriculum Design within an outcomes-based framework 
(how to help learners achieve results) 

Assessment and Evaluation of Learner Achievement 
(verification of the results of learning) 

Program Review 

(focus on accountability at the college and system-wide levels) 
Issues and Questions 



OUTCOMES-BASED EDUCATION IN THE COLLEGES: 
ITS IMPACT ON PROGRAM REVIEW AND THE EVALUATION OF LEARNERS 



1. Introduction to the Presentation 

This session is about one Ontario college's attempt to address system-wide educational reform 
initiatives to prepare for a "new economy". 

In March 1992, the College Standards and Acc Citation Council (CSAC) Esublishment 
Board which was created by the Minister of Education on recommendation of the Vision 
2000 Report (1990) released a Discussion Paper. This Paper proposed wide-ranging reform 
for college education and training within, the context of four major new initiatives: 

• general education 
generic skills standards 

• vocational program standards 
system-wide program review. 

In December 1991, Algonquin College had already begun to lay plans for a second cycle of 
program review that would provide ongoing formative evaluation for programs as well as 
continuous, updated information on major program indicators such as admissions demand, 
student retention, program costs and graduate placement 

The way in which the Algonquin initiative and CSAC came together to create a dynamic _ 
process of change and revitalization within Algonquin College is a story worth telling. This 
presentation today will tell you about some of the creative ways in which our College is 
planning to interface and articulate with the CSAC (Provincial) initiatives in order to ensure 
full compliance with system requirements and retain our local autonomy, ga ; n and maintain 
staff commitment to creative innovation and positive renewal, and foster greater allegiance to 
the College by our learner population. 

2. Ontario's "Vision" for the Colleges 

Plans for change and revitalization in college education in Ontario began in 1988 with the 
inauguration of the Vision 2000 Project which culminated in a Report entitled: Quality and 
O pportunity in June, 1990. The two components of a direction for change have been 
described as follows: 

"Vision 2000: Quality and Opportunity is about assuring the quality of career education 
provided by Ontario's college system and enhancing opportunities for students to participate. 
It is about how the colleges can help ensure that Ontario has the skilled and adaptable 
workforce necessary to prosper in the global economy, while contributing to the development 
of individuals as informed, productive and oocially responsible citizens." 
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3. The Role of Learning Outcomes (focus on performance standards) 



The CSAC initiatives rely on learning outcome* as the basis for the definition of program 
standards- The CSAC Establishment Board Report described the introduction of learning 
outcomes as a vehicle for addressing accountability as follows: 

'The goal of establishing system-wide outcome standards is to attempt, on a province-wide 
basis, to ensure similar outcomes for similar college programs...and to ensure the public that 
these standards are being met." 

A paper entitled "Guidelines to the Development of Standards of Achievement through 
Learning Outcomes" (January, 1994), defined learning outcomes this way: 

"Learning outcomes represent culminating demonstrations of learning and achievement They 
are not simply a listing of discrete skills, nor broad statements of knowledge and 
comprehension. They describe performances that demonstrate that significant learning has 
been verified and achieved by graduates of the program." 

The paper continues: "Readily amenable to system-wide application and review, with an 
increased emphasis on learner achievement of clearly defined performances, learning 
outcomes provide support for both of the fundamental goals of reform, namely, accountability 
and accessibility". 
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What are Learning Outcomes? 



Characteristics of Learning Outcomes 

Learning Outcomes: 

describe essential , significant meaningful, role performances that all college graduates 
should demonstrate to be effective in their vocational and personal lives 

are verifiable using a variety of assessment methods which may be quantitative 
measurement or qualitative reporting 

represent transferable abilities in that the role performances may be relevant to a range 
of work and/or life contexts 

do not dictate specific curriculum, Le. subject matter/content, teaching strategies or 
assessment methods 

must all be reliably demonstrated by all learners before graduation. 



How Are Learning Outcomes Determined? 



Learning Outcome statements for programs are determined through a process of consultation 
with employers, practitioners and educators. They da not simply reflect existing curriculum. 
The most likely steps in the development of learning outcomes for programs is as follows: 

1. Create a .profile of the effective graduate by listing the characteristics that are present 
in an individual who is most likely to succeed in a vocation. 

2. Envisage future trends in the vocational area, such as technological change, and their 
implications for the graduate working in the area. 

3. Postulate, the performances that graduates of a program should demonstrate in order to 
practise successfully in the vocation* 

4. Distill the information that has been generated in Steps 1-3 into a list by: 

a) grouping some items together 

b) assigning priority to items 

c) eliminating non-essential items 

d) adding forgotten items 

e) finalizing die list 

5. Prepare draft learning outcomes that are consistent with the essential characteristics of 
learning outcomes. 

6. Review and revise the learning outcomes by checking them against the criteria of 
essentifi characteristics. 

7. Submit the learning outcomes to a process of validation and review. 
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HOW ARE LEARNING OUTCOME STATEMENTS DRAFTED? 

Capture one essential performance at a time. 



Ensure that the learning outcome reflects a performance that is essential, significant 
and transferable. 



Ensure that the learning outcome is not stated too broadly. 



Ensure that the learning outcome is not stated too narrowly. 



Ensure that the learning outcome does not dictate curriculum. 



Ensure that the learning outcome describes post-secondary college level learning. 



Ensure that the wording of the learning outcome is clear. 



Ensure that the learning outcome is verifiable. 



LEARNING OUTCOMES FOR OUTCOMES-BASED EDUCATION 



Outcomes-based education is based on die notion that statements of the learning to be 
achieved by learners at the end of a unit of learning, ie. learning outcomes, serve as 
standards for assessment of learning. 

Program standards which include program learning outcome statements become criteria for 
determining the extent to which programs have met their obligations to learners, funding . 
agencies and the public. 

The concept of learning outcomes within outcomes-based education emphasizes the: 

• high performance of learners who graduate from our educational 
systems 

• importance of verifying results achieved at the end of a unit of learning 

• importance of accountability by our educational institutions. 

Learning outcon*s are the exit standards which serve as the performance criteria for 
evaluating learner achievement at the end of a learning unit or for assessing learning achieved 
through experience that is eligible for PLA credit 
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CURRICULUM DESIGN WITHIN AN OUTCOMES-BASED FRAMEWORK 



Write learning outcomes that identify significant, essential learning. 

♦ Use learning outcomes statements as exit standards - describing what all learners most 
achieve to receive credential. 

Design down from these exit outcomes. 

Create learning opportunities/activities that allow diverse learners to practice and 
eventually achieve learning outcomes. 

• ^ Plan assessment and evaluation that is directly related to learning outcomes. 
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ASSESSMENT AND EVALUATION OF LEARNER ACHIEVEMENT 

Verification of the Results of Learning 



IN OUTCOMES-BASED LEARNING MODELS, GRADUATES HAVE TO "RELIABLY 
DEMONSTRATE" ACHIEVEMENT OF THE LEARNING OUTCOME. 

"RELIABLY DEMONSTRATE" IMPLIES THAT LEARNERS WILL HAVE HAD 
OPPORTUNITIES TO PRACTISE, BE ASSESSED, AND RECEIVE FEEDBACK ON THEIR 
PROGRESS TOWARD EACH LEARNING OUTCOME. 

THEREFORE, IN ORDER TO GRANT CREDIT FOR ACHIEVEMENT OF A LEARNING 
OUTCOME, FACULTY WILL HAVE DOCUMENTED THE SUCCESSFUL PERFORMANCE OF 
THE EMBEDDED SKILLS AND KNOWLEDGE INHERENT IN A LEARNING OUTCOME AS 
WELL AS THE ROLE PERFORMANCE IDENTIFIED BY THE WHOLE LEARNING OUTCOME 
IN MORE THAN ONE OR TWO CONTEXTS. 
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PLANNING ASSESSMENT STRATEGIES FOR OUTCOMES-BASED EDUCATION 



A PLAN FOR THE ASSESSMENT OF LEARNERS SHOULD BE CLEARLY DESCRIBED. 
ASSESSMENT IS CONTINUOUS AND PROGRESSIVE. 

ANY INHERENT BIAS IN THE ASSESSMENT APPROACH SHOULD BE IDENTIFIED 
IN ADVANCE AND ELIMINATED AS MUCH AS POSSIBLE. 

AS LEARNERS PROGRESS TOWARD THE ACHIEVEMENT OF LEARNING 
OUTCOMES TEACHERS NEED TO PROVIDE FEEDBACK ON THEIR PROGRESS. 

EVALUATION OF PERFORMANCE OF A LEARNING OUTCOME WILL VERIFY THE 
LEARNER'S ABILITY TO INTEGRATE ALL ASPECTS OF THE PERFORMANCE, 
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EXAMPLES OF TYPES OF ASSESSMENT INSTRUMENTS OR METHODS 
observational recording and reporting 
self-evaluation 
peer assessment 
paper and pencil testing 
assignments and projects 
portfolios 

oral exams/interviews 
comprehensive testing 
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STANDARDS-REFERENCED EVALUATION 



EVALUATION OF LEARNER PERFORMANCE RELATED TO LEARNING OUTCOMES 
RELIES ON A CONFIGURATION OR PATTERN OF PERFORMANCE RELATED TO AN 
OUTCOME TAKEN OVER A SERIES OF TESTING/EVALUATION EPISODES IN 
WHICH THE LEARNER IS ASSESSED; 

EVALUATION CRITERIA ARE KNOWN IN ADVANCE BY THE LEARNER AND THE 
TEACHER; 

EVALUATION CRITERIA AND PERFORMANCE STANDARDS MAY CUT ACROSS A 
WIDE RANGE OF SUBJECT AREAS; 

EXPLICIT ASSESSMENT CRITERIA SHOULD BE COMMONLY UNDERSTOOD AND 
AGREED-UPON BY TEACHERS; 

SHOULD MAKE DIRECT AND EXTENSIVE USE OF TEACHERS' QUALITATIVE 
JUDGEMENTS; 

EVALUATION REPORTS REFLECT THE LEARNER'S SUCCESSFUL PERFORMANCE 
OF THE LEARNING OUTCOME; THEY MAY BE ANECDOTAL REPORTS OR CODES 
USED TO DESIGNATE THAT A LEARNING OUTCOME HAS BEEN ACHIEVED; 

IT IS NOT IMPORTANT TO RANK THE LEARNER ACCORDING TO THE 
PERFORMANCE OF CLASSMATES OR AGAINST A NORMATTVELY-BASED SCORE 
ESTABLISHED FOR A POPULATION WITH PARTICULAR CHARACTERISTICS AS IN 
STANDARDIZED TESTING; A LEARNER EITHER SUCCESSFULLY COMPLETES A 
LEARNING OUTCOME OR NOT; 

A PROGRAM OF STUDY IS FINISHED WHEN THE LEARNER HAS SUCCESSFULLY 
COMPLETED ALL LEARNING OUTCOMES ATTACHED TO A PROGRAM OR 
CREDENTIAL; 
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ASSESSMENT AND EVALUATION OF LEARNER 
ACHIEVEMENT OF LEARNING OUTCOMES 
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Implied learning/teaching methods: 



information/ 
knowledge through: 


skills through 
task performances 


demonstrated 
understanding 
through task or 
role performances 
with application 
in simulated 
contexts 


practise role 
performances in 
workplace or 
community 
settings 


e.g. lecture software/ 
CAI textbooks manuals 
workbooks 


e.g. assignments 
projects 
presentations 
in-class activities 
observation reports 


e.g. games 
simulations case 
studies/problem 
solving role play 
projects 


e.g. fieldwork 
ciinicals work 
placement 
internship 

work/life experience 



Assessment methods: 



tests 
quizzes 
examinations 
essays 

research papers 



challenge tests 



criterion-based 
scoring anecdotal 
feedback 



consolidation testing 
or assignments: 
e.g. comprehensive 
exams: oral, written 
or hands-on 



performance tests 

criterion-based 

scoring 



/ 



portfolio 



peer assessment or self - evaluation uping anecodotal or ratin gs methods 



observations/ 
anecdotal reporting 
performance testing 
journals/diaries 

portfolio 



portfolio 
preparation 
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Criterion referenced 



Criterion referenced 



standards referenced ; standards referenced 
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PROGRAM REVIEW 



(focus on accountability at the college and system-wide levels) 

Ontario's CSAC Report states that system-wide program review will address both fonnative and 
summative evaluation functions. It will also collect both quantitative and qualitative data about 
programs through an Inventory of Basic Descriptive Program Information with the use of a series of 
common indicators or instruments. 

Each college will be asked to submit a Critical Appraisal of its program under review which will 
include statistical information, evaluative comment drawn from several external and internal sources 
and indications of actions taken to meet any problems identified. 

A Provincial review panel appointed by CSAC will likely evaluate the Program Standards document, 
the Basic Inventory, data collected through Common Indicators and the Critical Appraisals. The Pane 
will determine if any follow-up is required such as additional information or a site visit. 

In addition to reviewing each program's Critical Appraisal, the Panel will evaluate the Provincial 
Program Standards documents in terms of their appropriateness and achievability by learners. 

Review panels may recommend one of the following for each college's program: 

• unconditional accreditation 

• conditional accreditation 

• accreditation withdrawn. 

Programs accredited without condition may not be reviewed until the next cycle (e.g. five years). 
Conditionally accredited programs may be reviewed every one or two years. 

Withdrawal of accreditation may lead to a recommendation that the program cease to receive 
Provincial funding. 
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FUNCTION AND PURPOSE OF PROGRAM REVIEW AT ALGONQUIN COLLEGE 
TO PROMOTE INSTITUTIONAL LEARNING AND RENEWAL 
TO GUIDE DECISION-MAKING 

TO PROMOTE ACCOUNT ABLITY THROUGH SYSTEMATIC SUMMATTVE 
EVALUATION REPORTS 

TO PROMOTE ACCOUNTABILITY THROUGH SYSTEMATIC 
FORMATIVE EVALUATION REPORTS 

TO PROMOTE RATIONAL, EQUITABLE AND EFFICIENT 
ALLOCATION OF RESOURCES 
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THE PROGRAM REVIEW SYSTEM 
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COMPONENTS OF THE PROGRAM REVIEW SYSTEM - OVERVIEW 



Critical Performance Indicators (CPFs) 

The CPrs: 

• are summative/quantitative 
are collected continuously 
include data related to: 

admissions demand 
attrition/retention 
program costing 
placement/employment 

Cyclical Program Evaluation 

rhis component 

follows a 5 year cycle 

obtains feedback from external stakesholders 
eg. graduates 
employers 

advisory comittee member? 
obtains feedback from students 

eg. graduating class exit survey - linked to learner achievement of teaming 
outcomes 

has survey results processed and tabulated centraiiy by Computer Services 
ensures that data analysis is supervised by the Planning Department 

Ongoing Program Appraisal 

The Program Review system intends to foster the continuing review and renewal of programs 
and courses offered by the College. The system should ensure that academic staff are 
supported in their regular examination and improvement of the courses and programs they 
deliver. 

The Ongoing Appraisal of courses and programs is ensured through: 

• systematic, regular Course Assessment, every course every term - linked to 
learner achievement of learning outcomes 
anual Program Self- Audit - linked to curriculum design to promote 
achievement of program learning outcomes 
Faculty Survey 

Program Review * 

The Interim Report and Action Plan report on the input derived from Cyclical Program 
Evaluation and Ongoing Program Appraisal and the Critical Performance Indicators. The 
Program Review Report outlines the revisions made to the program based on the Interim 
Report. It also reports on the present status of the program following program revision. 

Program Review Report 

Program Critical Appraisal (CSAC) 
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ISSUES AND QUESTIONS 

ISSUES: 

• outcomes-based versus time-based approaches to college level learning 

• resources needed to assist faculty in 'Curriculum planning to facilitate learner 
achievement of learning outcomes 

• resources and costs associated with cyclical program evaluation and ongoing program 
appraisal 

• grading practices - norm-referenced - to criterion-referenced - to standards- 
referenced??? 

• other issues: 
QUESTIONS: 

Conclusion* 
ACCC94 
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